System and method for display of multiple data channels on a single haptic display

ABSTRACT

A system that produces a haptic effect and generates a drive signal that includes at least two haptic effect signals each having a priority level. The haptic effect is a combination of the haptic effect signals and priority levels. The haptic effect may optionally be a combination of the two haptic effect signals if the priority levels are the same, otherwise only the haptic effect signal with the highest priority is used. The frequency of haptic notifications may also be used to generate a drive signal using foreground and background haptic effect channels depending on whether the frequency ratio exceeds a foreground haptic effect threshold.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority under 35 USC §120 tocopending application Ser. No. 14/134,140, filed Dec. 19, 2013, whichclaims the benefit of priority to application Ser. No. 13/782,825, U.S.Pat. No. 8,624,864, filed Mar. 1, 2013, which claims the benefit ofpriority to application Ser. No. 13/472,713, U.S. Pat. No. 8,570,296,filed May 16, 2012.

FIELD OF THE INVENTION

One embodiment is directed generally to a user interface for a device,and in particular to the display of multiple data channels of hapticfeedback for the user interface.

BACKGROUND INFORMATION

Electronic device manufacturers strive to produce a rich interface forusers. Conventional devices use visual and auditory cues to providefeedback to a user. In some interface devices, kinesthetic feedback(such as active and resistive force feedback) and/or tactile feedback(such as vibration, texture, and heat) is also provided to the user,more generally known collectively as “haptic feedback” or “hapticeffects”. Haptic feedback can provide cues that enhance and simplify theuser interface. Specifically, vibration effects, or vibrotactile hapticeffects, may be useful in providing cues to users of electronic devicesto alert the user to specific events, or provide realistic feedback tocreate greater sensory immersion within a simulated or virtualenvironment.

In order to generate vibration effects, many devices utilize some typeof actuator or haptic output device. Known haptic output devices usedfor this purpose include an electromagnetic actuator such as anEccentric Rotating Mass (“ERM”) in which an eccentric mass is moved by amotor, a Linear Resonant Actuator (“LRA”) in which a mass attached to aspring is driven back and forth, or a “smart material” such aspiezoelectric, electro-active polymers or shape memory alloys. Hapticoutput devices also broadly include non-mechanical or non-vibratorydevices such as those that use electrostatic friction (ESF), ultrasonicsurface friction (USF), or those that induce acoustic radiation pressurewith an ultrasonic haptic transducer, or those that use a hapticsubstrate and a flexible or deformable surface, or those that provideprojected haptic output such as a puff of air using an air jet, and soon.

Traditional architectures are designed to provide haptic feedback onlyfor a single haptic event. However, if multiple haptic events arecombined it may overwhelm or distract the user from a primary task.Therefore, there is a need for an improved system of providing a hapticeffect where low-importance or high-density information is perceivable,but not overwhelming or distracting from a primary task.

SUMMARY OF THE INVENTION

One embodiment is a system that produces a haptic effect and generates adrive signal that includes at least two haptic effect signals eachhaving a priority level. The haptic effect is a combination of thehaptic effect signals and priority levels. The haptic effect mayoptionally be a combination of the two haptic effect signals if thepriority levels are the same, otherwise only the haptic effect signalwith the highest priority is used. The frequency of haptic notificationsmay also be used to generate a drive signal using foreground andbackground haptic effect channels depending on whether the frequencyratio exceeds a foreground haptic effect threshold.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a haptically-enabled system according toone embodiment of the present invention.

FIG. 2 is a cut-away perspective view of an LRA implementation of ahaptic actuator according to one embodiment of the present invention.

FIG. 3 is a cut-away perspective view of an ERM implementation of ahaptic actuator according to one embodiment of the present invention.

FIGS. 4A-4C are views of a piezoelectric implementation of a hapticactuator according to one embodiment of the present invention.

FIG. 5 is a view of a haptic device using electrostatic friction (ESF)according to one embodiment of the present invention.

FIG. 6 is a view of a haptic device for inducing acoustic radiationpressure with an ultrasonic haptic transducer according to oneembodiment of the present invention.

FIG. 7 is a view of a haptic device using a haptic substrate andflexible or deformable surface according to one embodiment of thepresent invention.

FIGS. 8A-8B are views of a haptic device using ultrasonic surfacefriction (USF) according to one embodiment of the present invention.

FIGS. 9A-9D are screen views of example foreground and background hapticapplications according to one embodiment of the present invention.

FIGS. 10A-10B are display graphs of example multiple data channels ofhaptic feedback according to one embodiment of the present invention.

FIG. 11 is a flow diagram for displaying multiple data channels ofhaptic feedback for priority based haptic events according to oneembodiment of the present invention.

FIG. 12 is a flow diagram for displaying multiple data channels ofhaptic feedback for frequency based haptic events according to oneembodiment of the present invention.

DETAILED DESCRIPTION

As described below, a dynamic haptic effect refers to a haptic effectthat evolves over time as it responds to one or more input parameters.Dynamic haptic effects are haptic or vibrotactile effects displayed onhaptic devices to represent a change in state of a given input signal.The input signal can be a signal captured by sensors on the device withhaptic feedback, such as position, acceleration, pressure, orientation,or proximity, or signals captured by other devices and sent to thehaptic device to influence the generation of the haptic effect.

A dynamic effect signal can be any type of signal, but does notnecessarily have to be complex. For example, a dynamic effect signal maybe a simple sine wave that has some property such as phase, frequency,or amplitude that is changing over time or reacting in real timeaccording to a mapping schema which maps an input parameter onto achanging property of the effect signal. An input parameter may be anytype of input capable of being provided by a device, and typically maybe any type of signal such as a device sensor signal. A device sensorsignal may be generated by any means, and typically may be generated bycapturing a user gesture with a device. Dynamic effects may be veryuseful for gesture interfaces, but the use of gestures or sensors arenot necessarily required to create a dynamic signal.

One common scenario that does not involve gestures directly is definingthe dynamic haptic behavior of an animated widget. For example, when auser scrolls a list, it is not typically the haptification of thegesture that will feel most intuitive, but instead the motion of thewidget in response to the gesture. In the scroll list example, gentlysliding the list may generate a dynamic haptic feedback that changesaccording to the speed of the scrolling, but flinging the scroll bar mayproduce dynamic haptics even after the gesture has ended. This createsthe illusion that the widget has some physical properties and itprovides the user with information about the state of the widget such asits velocity or whether it is in motion.

A gesture is any movement of the body that conveys meaning or userintent. It will be recognized that simple gestures may be combined toform more complex gestures. For example, bringing a finger into contactwith a touch sensitive surface may be referred to as a “finger on”gesture, while removing a finger from a touch sensitive surface may bereferred to as a separate “finger off” gesture. If the time between the“finger on” and “finger off” gestures is relatively short, the combinedgesture may be referred to as “tapping”; if the time between the “fingeron” and “finger off” gestures is relatively long, the combined gesturemay be referred to as “long tapping”; if the distance between the twodimensional (x,y) positions of the “finger on” and “finger off” gesturesis relatively large, the combined gesture may be referred to as“swiping”; if the distance between the two dimensional (x,y) positionsof the “finger on” and “finger off” gestures is relatively small, thecombined gesture may be referred to as “smearing”, “smudging” or“flicking”. Any number of two dimensional or three dimensional simple orcomplex gestures may be combined in any manner to form any number ofother gestures, including, but not limited to, multiple finger contacts,palm or first contact, or proximity to the device. A gesture can also beany form of hand movement recognized by a device having anaccelerometer, gyroscope, or other motion sensor, and converted toelectronic signals. Such electronic signals can activate a dynamiceffect, such as shaking virtual dice, where the sensor captures the userintent that generates a dynamic effect.

FIG. 1 is a block diagram of a haptically-enabled system 10 according toone embodiment of the present invention. System 10 includes a touchsensitive surface 11 or other type of user interface mounted within ahousing 15, and may include mechanical keys/buttons 13. Internal tosystem 10 is a haptic feedback system that generates vibrations onsystem 10. In one embodiment, the vibrations are generated on touchsurface 11.

The haptic feedback system includes a processor 12. Coupled to processor12 is a memory 20 and an actuator drive circuit 16, which is coupled toa haptic actuator 18. Processor 12 may be any type of general purposeprocessor, or could be a processor specifically designed to providehaptic effects, such as an application-specific integrated circuit(“ASIC”). Processor 12 may be the same processor that operates theentire system 10, or may be a separate processor. Processor 12 candecide what haptic effects are to be played and the order in which theeffects are played based on high level parameters. In general, the highlevel parameters that define a particular haptic effect includemagnitude, frequency and duration. Low level parameters such asstreaming motor commands could also be used to determine a particularhaptic effect. A haptic effect may be considered dynamic if it includessome variation of these parameters when the haptic effect is generatedor a variation of these parameters based on a user's interaction.

Processor 12 outputs the control signals to drive circuit 16 whichincludes electronic components and circuitry used to supply actuator 18with the required electrical current and voltage to cause the desiredhaptic effects. System 10 may include more than one actuator 18, andeach actuator may include a separate drive circuit 16, all coupled to acommon processor 12. Memory device 20 can be any type of storage deviceor computer-readable medium, such as random access memory (RAM) orread-only memory (ROM). Memory 20 stores instructions executed byprocessor 12. Among the instructions, memory 20 includes an actuatordrive module 22 which are instructions that, when executed by processor12, generate drive signals for actuator 18 while also determiningfeedback from actuator 18 and adjusting the drive signals accordingly.The functionality of module 22 is discussed in more detail below. Memory20 may also be located internal to processor 12, or any combination ofinternal and external memory.

Touch surface 11 recognizes touches, and may also recognize the positionand magnitude or pressure of touches on the surface, such as the numberof touches, the size of the contact points, pressure, etc. The datacorresponding to the touches is sent to processor 12, or anotherprocessor within system 10, and processor 12 interprets the touches andin response generates haptic effect signals. Touch surface 11 may sensetouches using any sensing technology, including capacitive sensing,resistive sensing, surface acoustic wave sensing, pressure sensing,optical sensing, etc. Touch surface 11 may sense multi-touch contactsand may be capable of distinguishing multiple touches that occur at thesame time. Touch surface 11 may be a touchscreen that generates anddisplays images for the user to interact with, such as keys, dials,etc., or may be a touchpad with minimal or no images.

System 10 may be a handheld device, such as a cellular telephone, PDA,computer tablet, gaming console, etc. or may be any other type of devicethat provides a user interface and includes a haptic effect system thatincludes one or more ERMs, LRAs, electrostatic or other types ofactuators. The user interface may be a touch sensitive surface, or canbe any other type of user interface such as a mouse, touchpad,mini-joystick, scroll wheel, trackball, game pads or game controllers,etc. In embodiments with more than one actuator, each actuator may havea different output capability in order to create a wide range of hapticeffects on the device. Each actuator may be any type of haptic actuatoror a single or multidimensional array of actuators.

FIG. 2 is a cut-away side view of an LRA implementation of actuator 18in accordance to one embodiment. LRA 18 includes a casing 25, amagnet/mass 27, a linear spring 26, and an electric coil 28. Magnet 27is mounted to casing 25 by spring 26. Coil 28 is mounted directly on thebottom of casing 25 underneath magnet 27. LRA 18 is typical of any knownLRA. In operation, when current flows through coil 28 a magnetic fieldforms around coil 28 which in interaction with the magnetic field ofmagnet 27 pushes or pulls on magnet 27. One current flowdirection/polarity causes a push action and the other a pull action.Spring 26 controls the up and down movement of magnet 27 and has adeflected up position where it is compressed, a deflected down positionwhere it is expanded, and a neutral or zero-crossing position where itis neither compressed or deflected and which is equal to its restingstate when no current is being applied to coil 28 and there is nomovement/oscillation of magnet 27.

For LRA 18, a mechanical quality factor or “Q factor” can be measured.In general, the mechanical Q factor is a dimensionless parameter thatcompares a time constant for decay of an oscillating physical system'samplitude to its oscillation period. The mechanical Q factor issignificantly affected by mounting variations. The mechanical Q factorrepresents the ratio of the energy circulated between the mass andspring over the energy lost at every oscillation cycle. A low Q factormeans that a large portion of the energy stored in the mass and springis lost at every cycle. In general, a minimum Q factor occurs withsystem 10 is held firmly in a hand due to energy being absorbed by thetissues of the hand. The maximum Q factor generally occurs when system10 is pressed against a hard and heavy surface that reflects all of thevibration energy back into LRA 18.

In direct proportionality to the mechanical Q factor, the forces thatoccur between magnet/mass 27 and spring 26 at resonance are typically10-100 times larger than the force that coil 28 must produce to maintainthe oscillation. Consequently, the resonant frequency of LRA 18 ismostly defined by the mass of magnet 27 and the compliance of spring 26.However, when an LRA is mounted to a floating device (i.e., system 10held softly in a hand), the LRA resonant frequency shifts upsignificantly. Further, significant frequency shifts can occur due toexternal factors affecting the apparent mounting weight of LRA 18 insystem 10, such as a cell phone flipped open/closed or the phone heldtightly.

FIG. 3 is a cut-away perspective view of an ERM implementation ofactuator 18 according to one embodiment of the present invention. ERM 18includes a rotating mass 301 having an off-center weight 303 thatrotates about an axis of rotation 305. In operation, any type of motormay be coupled to ERM 18 to cause rotation in one or both directionsaround the axis of rotation 305 in response to the amount and polarityof voltage applied to the motor. It will be recognized that anapplication of voltage in the same direction of rotation will have anacceleration effect and cause the ERM 18 to increase its rotationalspeed, and that an application of voltage in the opposite direction ofrotation will have a braking effect and cause the ERM 18 to decrease oreven reverse its rotational speed.

One embodiment of the present invention provides haptic feedback bydetermining and modifying the angular speed of ERM 18. Angular speed isa scalar measure of rotation rate, and represents the magnitude of thevector quantity angular velocity. Angular speed or frequency w, inradians per second, correlates to frequency v in cycles per second, alsocalled Hz, by a factor of 2 φ. The drive signal includes a drive periodwhere at least one drive pulse is applied to ERM 18, and a monitoringperiod where the back electromagnetic field (“EMF”) of the rotating mass301 is received and used to determine the angular speed of ERM 18. Inanother embodiment, the drive period and the monitoring period areconcurrent and the present invention dynamically determines the angularspeed of ERM 18 during both the drive and monitoring periods.

FIGS. 4A-4C are views of a piezoelectric implementation of a hapticactuator 18 according to one embodiment of the present invention. FIG.4A shows a disk piezoelectric actuator that includes an electrode 401, apiezo ceramics disk 403 and a metal disk 405. As shown in FIG. 4B, whena voltage is applied to electrode 401, the piezoelectric actuator bendsin response, going from a relaxed state 407 to a transformed state 409.When a voltage is applied, it is that bending of the actuator thatcreates the foundation of vibration. Alternatively, FIG. 4C shows a beampiezoelectric actuator that operates similarly to a disk piezoelectricactuator by going from a relaxed state 411 to a transformed state 413.

FIG. 5 is a view of a haptic device using electrostatic friction (ESF)according to one embodiment of the present invention. Similar to theoperational principles described by Makinen et al. in U.S. Pat. No.7,982,588, the embodiment is based on the discovery that subcutaneousPacinian corpuscles can be stimulated by means of a capacitiveelectrical coupling and an appropriately dimensioned control voltage,either without any mechanical stimulation of the Pacinian corpuscles oras an additional stimulation separate from such mechanical stimulation.An appropriately dimensioned high voltage is used as the controlvoltage. In the present context, a high voltage means such a voltagethat direct galvanic contact must be prevented for reasons of safetyand/or user comfort. This results in a capacitive coupling between thePacinian corpuscles and the apparatus causing the stimulation, whereinone side of the capacitive coupling is formed by at least onegalvanically isolated electrode connected to the stimulating apparatus,while the other side, in close proximity to the electrode, is formed bythe body member, preferably a finger, of the stimulation target, such asthe user of the apparatus, and more specifically the subcutaneousPacinian corpuscles.

It likely that the invention is based on a controlled formation of anelectric field between an active surface of the apparatus and the bodymember, such as a finger, approaching or touching it. The electric fieldtends to give rise to an opposite charge on the proximate finger. Alocal electric field and a capacitive coupling can be formed between thecharges. The electric field directs a force on the charge of the fingertissue. By appropriately altering the electric field a force capable ofmoving the tissue may arise, whereby the sensory receptors sense suchmovement as vibration.

As shown in FIG. 5, one or more conducting electrodes 501 are providedwith an insulator. When a body member such as finger 505 is proximate tothe conducting electrode 501, the insulator prevents flow of directcurrent from the conducting electrode to the body member 505. Acapacitive coupling field force 503 over the insulator is formed betweenthe conducting electrode 501 and the body member 505. The apparatus alsocomprises a high-voltage source for applying an electrical input to theone or more conducting electrodes, wherein the electrical inputcomprises a low-frequency component in a frequency range between 10 Hzand 1000 Hz. The capacitive coupling and electrical input aredimensioned to produce an electrosensory sensation which is producedindependently of any mechanical vibration of the one or more conductingelectrodes or insulators.

FIG. 6 is a view of a haptic device for inducing acoustic radiationpressure with an ultrasonic haptic transducer similar to that describedby Iwamoto et al., “Non-contact Method for Producing Tactile SensationUsing Airborne Ultrasound”, Eurohaptics 2008, LNCS 5024, pp. 504-513. Anairborne ultrasound transducer array 601 is designed to provide tactilefeedback in three-dimensional (3D) free space. The array radiatesairborne ultrasound, and produces high-fidelity pressure fields onto theuser's hands without the use of gloves or mechanical attachments. Themethod is based on a nonlinear phenomenon of ultrasound; acousticradiation pressure. When an object interrupts the propagation ofultrasound, a pressure field is exerted on the surface of the object.This pressure is called acoustic radiation pressure. The acousticradiation pressure P [Pa] is simply described as P=αE, where E[J=m³] isthe energy density of the ultrasound and a is a constant ranging from 1to 2 depending on the reflection properties of the surface of theobject. The equation describes how the acoustic radiation pressure isproportional to the energy density of the ultrasound. The spatialdistribution of the energy density of the ultrasound can be controlledby using the wave field synthesis techniques. With an ultrasoundtransducer array, various patterns of pressure field are produced in 3Dfree space. Unlike air-jets, the spatial and temporal resolutions arequite fine. The spatial resolution is comparable to the wavelength ofthe ultrasound. The frequency characteristics are sufficiently fine upto 1 kHz.

The airborne ultrasound can be applied directly onto the skin withoutthe risk of the penetration. When the airborne ultrasound is applied onthe surface of the skin, due to the large difference between thecharacteristic acoustic impedance of the air and that of the skin, about99.9% of the incident acoustic energy is reflected on the surface of theskin. Hence, this tactile feedback system does not require the users towear any clumsy gloves or mechanical attachments.

FIG. 7 shows a three-dimensional (3D) diagram illustrating a hapticdevice 701 using a haptic substrate and a flexible surface in accordancewith one embodiment of the present invention. Device 701 includes aflexible surface layer 703, a haptic substrate 705, and a deformingmechanism 711. It should be noted that device 701 can be a userinterface device, such as an interface for a cellular phone, a personaldigital assistant (“PDA”), an automotive data input system, and soforth. It should be further noted that the underlying concept of theexemplary embodiment of the present invention would not change if one ormore blocks (circuits or layers) were added to or removed from device701.

Flexible surface layer 703, in one instance, is made of soft and/orelastic materials such as silicone rubber, which is also known aspolysiloxane. A function of the flexible surface layer 703 is to changeits surface shape or texture upon contact with the physical pattern ofhaptic substrate 705. The physical pattern of haptic substrate 705 isvariable as one or more of the local features 110-124 can be raised orlowered to present features to affect the surface of the flexiblesurface layer 703 upon contact. Once the physical pattern of hapticsubstrate 705 is determined, the texture of flexible surface layer 703can change to confirm its surface texture to the physical pattern ofhaptic substrate 705. It should be note that the deformation of flexiblesurface layer 703 from one texture to another can be controlled bydeforming mechanism 711. For example, when deforming mechanism 711 isnot activated, flexible surface layer 703 maintains its smoothconfiguration floating or sitting over haptic substrate 705. The surfaceconfiguration of flexible surface layer 703, however, deforms or changesfrom a smooth configuration to a coarse configuration when deformingmechanism 711 is activated and the haptic substrate 705 is in contactwith the flexible surface layer 703 so as to generate a similar patternon the top surface of the flexible surface layer 703.

Alternatively, flexible surface layer 703 is a flexible touch sensitivesurface, which is capable of accepting user inputs. The flexible touchsensitive surface can be divided into multiple regions wherein eachregion of the flexible touch sensitive surface can accept an input whenthe region is being touched or depressed by a finger. In one embodiment,the flexible touch sensitive surface includes a sensor, which is capableof detecting a nearby finger and waking up or turning on the device.Flexible surface layer 703 may also include a flexible display, which iscapable of deforming together with flexible surface layer 703. It shouldbe noted that various flexible display technologies can be used tomanufacture flexible displays, such as organic light-emitting diode(OLED), organic, or polymer TFT (Thin Film Transistor).

Haptic substrate 705 is a surface reconfigurable haptic device capableof changing its surface pattern in response to one or more patternactivating signals. Haptic substrate 705 can also be referred to as ahaptic mechanism, a haptic layer, a tactile element, and the like.Haptic substrate 705, in one embodiment, includes multiple tactile orhaptic regions 707, 709, wherein each region can be independentlycontrolled and activated. Since each tactile region can be independentlyactivated, a unique surface pattern of haptic substrate 705 can becomposed in response to the pattern activating signals. In anotherembodiment, every tactile region is further divided into multiple hapticbits wherein each bit can be independently excited or activated ordeactivated.

Haptic substrate 705, or a haptic mechanism, in one embodiment, isoperable to provide haptic feedback in response to an activating commandor signal. Haptic substrate 705 provides multiple tactile or hapticfeedbacks wherein one tactile feedback is used for surface deformation,while another tactile feedback is used for input confirmation. Inputconfirmation is a haptic feedback to inform a user about a selectedinput. Haptic mechanism 705, for example, can be implemented by varioustechniques including vibration, vertical displacement, lateraldisplacement, push/pull technique, air/fluid pockets, local deformationof materials, resonant mechanical elements, piezoelectric materials,micro-electro-mechanical systems (“MEMS”) elements, thermal fluidpockets, MEMS pumps, variable porosity membranes, laminar flowmodulation, or the like.

Haptic substrate 705, in one embodiment, is constructed by semi-flexibleor semi-rigid materials. In one embodiment, haptic substrate should bemore rigid than flexible surface 703 thereby the surface texture offlexible surface 703 can confirm to the surface pattern of hapticsubstrate 705. Haptic substrate 705, for example, includes one or moreactuators, which can be constructed from fibers (or nanotubes) ofelectroactive polymers (“EAP”), piezoelectric elements, fiber of shapememory alloys (“SMAs”) or the like. EAP, also known as biologicalmuscles or artificial muscles, is capable of changing its shape inresponse to an application of voltage. The physical shape of an EAP maybe deformed when it sustains large force. EAP may be constructed fromElectrostrictive Polymers, Dielectric elastomers, Conducting Polyers,Ionic Polymer Metal Composites, Responsive Gels, Bucky gel actuators, ora combination of the above-mentioned EAP materials.

SMA (Shape Memory Alloy), also known as memory metal, is another type ofmaterial which can be used to construct haptic substrate 705. SMA may bemade of copper-zinc-aluminum, copper-aluminum-nickel, nickel-titaniumalloys, or a combination of copper-zinc-aluminum,copper-aluminum-nickel, and/or nickel-titanium alloys. A characteristicof SMA is that when its original shape is deformed, it regains itsoriginal shape in accordance with the ambient temperature and/orsurrounding environment. It should be noted that the present embodimentmay combine the EAP, piezoelectric elements, and/or SMA to achieve aspecific haptic sensation.

Deforming mechanism 711 provides a pulling and/or pushing force totranslate elements in the haptic substrate 705 causing flexible surface703 to deform. For example, when deforming mechanism 711 creates avacuum between flexible surface 703 and haptic substrate 705, flexiblesurface 703 is pushed against haptic substrate 705 causing flexiblesurface 703 to show the texture of flexible surface 703 in accordancewith the surface pattern of haptic substrate 705. In other words, once asurface pattern of haptic substrate 705 is generated, flexible surfaceis pulled or pushed against haptic substrate 705 to reveal the patternof haptic substrate 705 through the deformed surface of flexible surface703. In one embodiment, haptic substrate 705 and deforming mechanism 711are constructed in the same or substantially the same layer.

Upon receipt of a first activating signal, haptic substrate 705generates a first surface pattern. After formation of the surfacepattern of haptic substrate 705, deforming mechanism 711 is subsequentlyactivated to change surface texture of flexible surface 703 in responseto the surface pattern of haptic substrate 705. Alternatively, if hapticsubstrate 705 receives a second activating signal, it generates a secondpattern.

Haptic substrate 705 further includes multiple tactile regions whereineach region can be independently activated to form a surface pattern ofthe substrate. Haptic substrate 705 is also capable of generating aconfirmation feedback to confirm an input selection entered by a user.Deforming mechanism 711 is configured to deform the surface texture offlexible surface 703 from a first surface characteristic to a secondsurface characteristic. It should be noted that haptic device furtherincludes a sensor, which is capable of activating the device when thesensor detects a touch on flexible surface 703. Deforming mechanism 711may be a vacuum generator, which is capable of causing flexible surface703 to collapse against the first surface pattern to transform itssurface configuration in accordance with the configuration of firstpattern of haptic substrate 705.

Haptic substrate 705 illustrates the state when tactile regions 707 and709 are activated. Tactile regions 707 and 709 are raised in a z-axisdirection. Upon receipt of one or more activating signals, hapticsubstrate 705 identifies a surface pattern in accordance with theactivating signals. Haptic substrate 705 provides identified pattern byactivating various tactile regions such as regions 707 and 709 togenerate the pattern. It should be noted that tactile regions 707 and709 imitate two buttons or keys. In another embodiment, tactile region707 or 709 includes multiple haptic bits wherein each bit can becontrolled for activating or deactivating.

FIG. 8 is a view of a haptic device using ultrasonic surface friction(USF) similar to that described by Biet et al., “New Tactile DevicesUsing Piezoelectric Actuators”, ACTUATOR 2006, 10^(th) InternationalConference on New Actuators, 14-16 Jun. 2006, Bremen, Germany. Anultrasonic vibration display 801 produces ultrasonic vibrations in theorder of a few micrometers. The display 801 consists of a touchinterface surface 803 that vibrates at the ultrasound range. Thevibrations 805 travel along the touch surface 803 at a speed v_(t) whena finger 809 is in contact and applies a force 807 F_(t) to the surface803. The vibrations 805 create an apparent reduction of friction on thesurface 803. One explanation is that by moving up and down, the touchsurface 803 creates an air gap 813 between the surface 803 and theinteracting finger 809, and is the air gap 813 that causes the reductionin friction. This can be thought as of a Lamb wave 815 along the surface803 that at some instants in time is in contact with the finger 809 whenthe finger 809 is in contact with the crest or peak of the wave 805, andsometimes is not when the finger 809 is above the valley of the wave805. When finger 809 is moved in a lateral direction 811 at a speedv_(t), the apparent friction of the surface 803 is reduced due to the onand off contact of the surface 803 with the finger 809. When the surface803 is not activated, the finger 809 is always in contact with thesurface 803 and the static or kinetic coefficients of friction remainconstant.

Because the vibrations 805 occur on surface 803 in the ultrasound rangeof typically 20 KHz or greater, the wavelength content is usuallysmaller than the finger size, thus allowing for a consistent experience.It will be noted that the normal displacement of surface 803 is in theorder of less than 5 micrometers, and that a smaller displacementresults in lower friction reduction.

FIGS. 9A-9D are screen views of example foreground and background hapticapplications according to one embodiment of the present invention. Itwill be recognized that more than one haptic enabled softwareapplication may be running simultaneously on a device having a hapticactuator, and that a window on the top of a virtual windows environmentmay overlap or obscure portions of any windows that are on the bottom.FIG. 9A shows a screen view of an example application window having avirtual download application button located in the center of the screen.In FIG. 9B the user selects the download application button, whereuponFIG. 9C shows a new screen view having a status bar in the center of thescreen which indicates the percentage completion of the download. Thestatus bar changes color proportionally from left to right correspondingto the percentage completion text shown directly below the status bar.Because the status bar is haptified, a haptic effect signal is generatedand output to the haptic actuator concurrently with the visual displayof the status bar. In one embodiment, the haptic effect signal changesover time corresponding to the percentage completion of the download.

FIG. 9D shows a screen view of a text input window. The text inputwindow, selected by the user as the active window, is running in theforeground and completely obscures the download application status barwhich is running simultaneously in the background. Although the downloadapplication window is no longer the active window and the status bar iscompletely obscured on the visual display, the status bar haptic effectsignal continues to be generated and output to the haptic actuator as abackground haptic effect. Because the text input window is alsohaptified, a foreground haptic effect signal is generated and output tothe haptic actuator for each typed character concurrently with thevisual display of the typed character in the text input window. In oneembodiment, the foreground and background haptic effect signals arecombined, modified or synthesized in such a way that the user perceivesthe foreground and background haptic effects as being distinct hapticeffects even though they are both being output concurrently via a singlehaptic actuator.

The perception of a haptic effect has three different levels. The firstlevel is the threshold of perception, which is the minimum appliedhaptic effect signal component or components required for a user todetect the haptic effect. Such haptic components include, but are notlimited to, strength, frequency, duration, rhythm and dynamics of thehaptic effect signal. It will be recognized that the threshold of hapticperception may be highly non-linear and may vary greatly between users,and may even vary for a single user depending on many factors such asthe user's sensitivity to touch, how tightly the user may be holding ahandheld device, the ambient temperature, the user's age, or the user'sphysical activity or environment such as walking or riding in a vehicle,and so on.

The second level of haptic effect perception is the threshold ofattention break-in, which is the minimum change in the applied hapticeffect signal that results in drawing the user's attention away from theprimary focus to the attention break-in haptic effect itself. It will berecognized that the threshold of attention break-in may vary betweenusers or for a single user depending on many factors as described above,and may also vary depending on whether the attention break-in is relatedto various types of haptic effects including a positive additive effect,or a negative subtractive effect, or a change in the haptic effect. Thethird level of haptic effect perception is the threshold of pain, whichalso varies between users or for a single user depending on many factorsas described above. It will be recognized that under some circumstances,the threshold of perception may be the same as the threshold ofattention break-in, which may also be the same as the threshold of pain.

The present invention is compatible with a wide variety of hapticactuators, and can present multiple channels of haptic effect data withdifferent intensity levels. In one embodiment, the multiple channels arerepresented by a foreground channel and one or more background channels.A background haptic effect is any haptic effect or haptic effectcomponent which meets or exceeds the threshold of perception. Aforeground haptic effect is any haptic effect or haptic effect componentwhich meets or exceeds the threshold of attention break-in. In oneembodiment, a foreground or background haptic effect may be a definedset of static or dynamic haptic effects or effect components. In anotherembodiment, a foreground or background haptic effect may be an adaptiveset of static or dynamic haptic effects or haptic effect components inresponse to user input, system input, device sensor input or ambientinput.

Using multiple haptic channels, such as foreground and backgroundchannels, enables subtle haptic effects to be provided concurrently withmore obvious haptic effects, allowing a user to distinguish between thedifferent effects and identifying them as originating from differentsources. In one embodiment, low-importance or high-density informationis perceivable, but not overwhelming or distracting from a primary task,and multiple channels further enable haptic ambient awareness. Forexample, a haptic enabled handheld or mobile device which is monitoringthe local weather during a rainstorm activates a background hapticchannel to provide a sensation of raindrops that increases or decreasesas it rains harder or softer.

In one embodiment, foreground and background channels are used todistinguish the feedback originating from a local device and thefeedback originating from another user. For example, a messagenotification arriving from another user activates a foreground hapticeffect, while the status of a ticking clock on the local deviceactivates a background haptic effect.

In one embodiment, foreground and background channels are used todistinguish the feedback originating from a local device and thefeedback originating from a primary user. For example, the feedbackoriginated by a primary user typing on a haptic enabled keyboardactivates a foreground haptic effect, while the status of a progress baron the local device activates a background haptic effect.

In one embodiment, foreground and background channels are used todistinguish the feedback within or between virtual simulations oranimations. For example, the motion of a virtual rolling ball activatesa foreground haptic effect, while the virtual texture the ball isrolling on activates a background haptic effect.

In one embodiment, background haptic effects are additive such that whenmultiple background effects are received concurrently or in quicksuccession, the overall result is a natural or gradual foregrounding ofthe haptic effects. For example, a single background text message“tweet” notification received from a non-primary user may be easilymissed or ignored by the primary user, but when hundreds or thousands ofmessage notifications constituting a “tweet storm” are received in ashort amount of time, the multiple haptic effects add up and the overallresult is a haptic experience in the foreground which draws the primaryuser's attention to the event.

In one embodiment, background haptic effects are used to providenon-distracting or “polite” augmentation of a commercial advertisementor any other type of haptic encoded content. For example, anadvertisement for a carbonated soft drink provides a background haptic“fizz” effect that can be felt if the user is paying attention butotherwise can be easily ignored.

It will be recognized that any type of input such as user, device,system, application or network input may be represented by any number ofhaptic events on one or more foreground or background haptic channels.Examples include, but are not limited to, multi-tasking applications,incoming email, “tweet” message notifications, passive notifications,outgoing messages, progress bars, Bluetooth or local device pairings,network add or drop connection, continuous antenna signal level, and soon.

FIGS. 10A-10B are display graphs of example multiple data channels ofhaptic feedback according to one embodiment of the present invention.FIG. 10A shows a graph of the perceptual magnitude of a haptic signalover time for priority based haptic events, along with a correspondinggraph of notification activity. At time T1, the perceptual magnitude ofa haptic signal 1001 corresponding to the medium priority notificationsN1 and N2 starts in the background channel 1003. Upon receipt of a highpriority notification N3, at time T2 the haptic signal 1001 begins torise until at time T3 the haptic signal 1001 crosses the threshold fromthe background channel 1003 into the foreground channel 1005. The hapticsignal 1001 continues to increase up to a peak level 1007, where in theabsence of any further notifications the haptic signal 1001 decreasesand crosses the threshold from the foreground channel 1005 to thebackground channel 1003 at time T4.

At time T5, receipt of a high priority notification once again causesthe haptic signal 1001 to rise until at time T6 the haptic signalcrosses 1001 the threshold from the background channel 1003 into theforeground channel 1005. The haptic signal 1001 continues to increase upto a peak level 10010, where in the absence of any further notificationsthe haptic signal 1001 decreases and crosses the threshold from theforeground channel 1005 to the background channel 1003 at time T7. Itwill be recognized that a stream of low-priority or medium-prioritynotifications punctuated with high-priority notifications results in ahaptic signal 1001 that shifts between the background channel 1003 andforeground channel 1005 without limitation.

FIG. 10B shows a graph of the perceptual magnitude of a haptic signalover time for frequency based haptic events, along with a correspondinggraph of notification activity. At time T8, the perceptual magnitude ofa haptic signal 1011 corresponding to the relatively infrequentnotifications N1 through N3 starts in the background channel 1013. Uponreceipt of higher frequency notifications, at time T9 the haptic signal1011 begins to rise until at time T10 the haptic signal 1011 crosses thethreshold from the background channel 1013 into the foreground channel1015. With continuing receipt of higher frequency notifications, thehaptic signal 1011 continues to increase up to a peak level 1017, wherein the absence of any further notifications the haptic signal 1011decreases and crosses the threshold from the foreground channel 1015 tothe background channel 1013 at time T11. It will be recognized that astream of low-frequency notifications punctuated with high-frequencynotifications results in a haptic signal 1011 that shifts between thebackground channel 1013 and foreground channel 1015 without limitation.In one embodiment, priority based haptic events and frequency basedhaptic events may be interspersed with each other or received at anytime or in any order, and may be used in any manner to generate anoverall combined haptic signal.

FIG. 11 is a flow diagram for displaying multiple data channels ofhaptic feedback for priority based haptic events according to oneembodiment of the present invention. In one embodiment, thefunctionality of the flow diagram of FIG. 11 is implemented by softwarestored in memory or other computer readable or tangible medium, andexecuted by a processor. In other embodiments, the functionality may beperformed by hardware (e.g., through the use of an application specificintegrated circuit (“ASIC”), a programmable gate array (“PGA”), a fieldprogrammable gate array (“FPGA”), etc.), or any combination of hardwareand software.

At 1101, the system receives input of first and second haptic effectsignals having first and second priority levels. It will be recognizedthat any type or number of priority levels may be used, such asforeground and background priority levels, or any number ofalpha-numeric or any other sequential or non-sequential priority levels,without limitation. The first and second haptic effect signals may bereceived in any order or time sequence, either sequentially withnon-overlapping time periods or in parallel with overlapping orconcurrent time periods. At 1103, the system compares the first prioritylevel to the second priority level. If at 1105 the first priority levelis less than the second priority level, at 1107 an interaction parameteris generated using the second haptic signal. It will be recognized thatany type of input synthesis method may be used to generate theinteraction parameter from one or more haptic effect signals including,but not limited to, the method of synthesis examples listed in TABLE 1below. If at 1109 the first priority level is equal to the secondpriority level, at 1111 an interaction parameter is generated using thesecond haptic signal. If at 1113 the first priority level is greaterthan the second priority level, at 1115 an interaction parameter isgenerated using the second haptic signal. At 1117, a drive signal isapplied to a haptic actuator according to the interaction parameter.

Table 1

Methods of Synthesis

-   -   Additive synthesis—combining inputs, typically of varying        amplitudes    -   Subtractive synthesis—filtering of complex signals or multiple        signal inputs    -   Frequency modulation synthesis—modulating a carrier wave signal        with one or more operators    -   Sampling—using recorded inputs as input sources subject to        modification    -   Composite synthesis—using artificial and sampled inputs to        establish a resultant “new” input    -   Phase distortion—altering the speed of waveforms stored in        wavetables during playback    -   Waveshaping—intentional distortion of a signal to produce a        modified result    -   Resynthesis—modification of digitally sampled inputs before        playback    -   Granular synthesis—combining of several small input segments        into a new input    -   Linear predictive coding—similar technique as used for speech        synthesis    -   Direct digital synthesis—computer modification of generated        waveforms    -   Wave sequencing—linear combinations of several small segments to        create a new input    -   Vector synthesis—technique for fading between any number of        different input sources    -   Physical modeling—mathematical equations of the physical        characteristics of virtual motion

FIG. 12 is a flow diagram for displaying multiple data channels ofhaptic feedback for frequency based haptic events according to oneembodiment of the present invention. At 1201, the system receives one ormore haptic effect notifications N over a non-zero time period T. At1203, the system generates a notification frequency ratio R, calculatedby using at least the number of haptic effect notifications N and thenon-zero time period T. In one embodiment, the notification frequencyratio R is calculated as N divided by T. At 1205, the system comparesthe notification frequency ratio R to a foreground haptic threshold F.Haptic threshold F may be static or dynamic and may vary over timedepending on many factors such as the user's sensitivity to touch, howtightly the user may be holding a handheld device, the ambienttemperature, the user's age, or the user's physical activity orenvironment such as walking or riding in a vehicle, and so on. It willbe recognized that the notification frequency ratio R may be directlycalculated or may be normalized corresponding to a wide range ofvariation for the haptic threshold F, and that the haptic threshold Fmay be directly calculated or may be normalized corresponding to a widerange of variation for the notification frequency ratio R.

If at 1207 the notification frequency ratio R is less than theforeground haptic threshold F, at 1209 an interaction parameter isgenerated using a background haptic signal. If at 1211 the notificationfrequency ratio R is greater than or equal to the foreground hapticthreshold F, at 1213 an interaction parameter is generated using aforeground haptic signal. At 1215, a drive signal is applied to a hapticactuator according to the interaction parameter

Several embodiments are specifically illustrated and/or describedherein. However, it will be appreciated that modifications andvariations of the disclosed embodiments are covered by the aboveteachings and within the purview of the appended claims withoutdeparting from the spirit and intended scope of the invention.

What is claimed:
 1. A method of producing a haptic effect comprising:receiving a first input signal from a virtual simulation or animation;receiving a second input signal from the virtual simulation oranimation; if the second input signal has a lower priority than thefirst input signal, generating a dynamic interaction parameter using thefirst input signal, the second input signal, and a haptic levelcorresponding to a threshold of perception; if the second input signalhas a higher priority than the first input signal, generating a dynamicinteraction parameter using the first input signal, the second inputsignal, and a haptic level corresponding to a threshold of attentionbreak-in; applying a drive signal to a haptic output device according tothe dynamic interaction parameter allowing a user to distinguish betweenthe first input signal and the second input signal.
 2. The method ofclaim 1, wherein determining the haptic level comprises using a signalcomponent selected from the list consisting of strength, frequency,duration, rhythm, or dynamics.
 3. The method of claim 1, wherein thefirst input signal or the second input signal comprises feedbackoriginating from a local device.
 4. The method of claim 1, wherein thefirst input signal or the second input signal comprises feedbackoriginating from a primary user.
 5. The method of claim 1, wherein thefirst input signal or the second input signal comprises feedbackoriginating from another user.
 6. The method of claim 1, wherein thehaptic level comprises a defined or adaptive set of haptic effects. 7.The method of claim 1, wherein the threshold of perception comprises anatural or gradual foregrounding threshold.
 8. The method of claim 1,wherein the threshold of perception comprises a non-distracting or“polite” threshold.
 9. The method of claim 1, wherein the threshold ofattention break-in comprises a threshold of pain.
 10. The method ofclaim 1, wherein a virtual simulation or animation comprises amulti-tasking application.
 11. A haptic effect enabled devicecomprising: a haptic output device; a drive module electronicallycoupled to the haptic output device for receiving a first input signalfrom a virtual simulation or animation, and receiving a second inputsignal from the virtual simulation or animation, and if the second inputsignal has a lower priority than the first input signal, generating adynamic interaction parameter using the first input signal, the secondinput signal, and a haptic level corresponding to a threshold ofperception, and if the second input signal has a higher priority thanthe first input signal, generating a dynamic interaction parameter usingthe first input signal, the second input signal, and a haptic levelcorresponding to a threshold of attention break-in; and a drive circuitelectronically coupled to the drive module and the haptic output devicefor applying a drive signal to a haptic output device on the deviceaccording to the dynamic interaction parameter allowing a user todistinguish between the first input signal and the second input signal.12. The system of claim 11, wherein the drive module comprises a drivemodule for determining the first haptic level or the second haptic levelcomprises using a signal component selected from the list consisting ofstrength, frequency, duration, rhythm, or dynamics.
 13. The system ofclaim 11, wherein the first input signal or the second input signalcomprises feedback originating from a local device.
 14. The system ofclaim 11, wherein the first input signal or the second input signalcomprises feedback originating from a primary user.
 15. The system ofclaim 11, wherein the first input signal or the second input signalcomprises feedback originating from another user.
 16. The system ofclaim 11, wherein the haptic level comprises a defined or adaptive setof haptic effects.
 17. The system of claim 11, wherein the threshold ofperception comprises a natural or gradual foregrounding threshold. 18.The system of claim 11, wherein the threshold of perception comprises anon-distracting or “polite” threshold.
 19. The system of claim 11,wherein the threshold of attention break-in comprises a threshold ofpain.
 20. The system of claim 11, wherein a virtual simulation oranimation comprises a multi-tasking application.
 21. A non-transitorycomputer readable medium having instructions stored thereon that, whenexecuted by a processor, causes the processor to produce a hapticeffect, the instructions comprising: receiving a first input signal froma virtual simulation or animation; receiving a second input signal fromthe virtual simulation or animation; if the second input signal has alower priority than the first input signal, generating a dynamicinteraction parameter using the first input signal, the second inputsignal, and a haptic level corresponding to a threshold of perception;if the second input signal has a higher priority than the first inputsignal, generating a dynamic interaction parameter using the first inputsignal, the second input signal, and a haptic level corresponding to athreshold of attention break-in; applying a drive signal to a hapticoutput device according to the dynamic interaction parameter allowing auser to distinguish between the first input signal and the second inputsignal.
 22. The non-transitory computer readable medium of claim 21,wherein determining the haptic level comprises using a signal componentselected from the list consisting of strength, frequency, duration,rhythm, or dynamics.
 23. The non-transitory computer readable medium ofclaim 21, wherein the first input signal or the second input signalcomprises feedback originating from a local device.
 24. Thenon-transitory computer readable medium of claim 21, wherein the firstinput signal or the second input signal comprises feedback originatingfrom a primary user.
 25. The non-transitory computer readable medium ofclaim 21, wherein the first input signal or the second input signalcomprises feedback originating from another user.
 26. The non-transitorycomputer readable medium of claim 21, wherein the haptic level comprisesa defined or adaptive set of haptic effects.
 27. The non-transitorycomputer readable medium of claim 21, wherein the threshold ofperception comprises a natural or gradual foregrounding threshold. 28.The non-transitory computer readable medium of claim 21, wherein thethreshold of perception comprises a non-distracting or “polite”threshold.
 29. The non-transitory computer readable medium of claim 21,wherein the threshold of attention break-in comprises a threshold ofpain.
 30. The non-transitory computer readable medium of claim 21,wherein a virtual simulation or animation comprises a multi-taskingapplication.